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ABSTRACT : 

A ground-based simulator facility employing a two-axis 
compensatory tracking task with a random appearing signal 
was used to evaluate the performance of one hundred five 
pilot and non-pilot test subjects using four separate con- 
trol sticks -- two moveable and two rigid. Pilot acceptance 
of the rigid cockpit controllers was determined by compar- 
ing individual pilot ratings of the sticks. In general, in 
both performance and opinion, the rigid systems were found 
to be superior to their moveable counterparts. Steps were 
taken to avoid errors due to pilot bias, learning, fatigue, 
or adapation. The results obtained are subject to several 
test limitations, including the low stick-force levels em- 
ployed, the lack of aircraft vibration effects, and the 
realism of the simulation. 
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I. INTRODUCTION 



It has been traditional for aircraft to have cockpit 
control sticks that move in a certain direction a given 
amount in order to impart movement to the control surfaces. 

This type of control has evolved from the low complexity 
system where the control stick is directly connected to the 
control surfaces (reversible control) so that stick deflect- 
ion is a direct measure of control surface deflection. On 
modern, high-speed aircraft, however, the forces required 
to move the control surfaces may exceed the physical cap- 
ability of a human pilot and some form of powered or power- 
assisted control system is necessary. In fully powered con- 
trol systems, there is no direct force or position coupling 
between the pilot's control stick and the control surfaces, 
and any cockpit indications must come from 'artificial' 
feedback signals. 

As the simple mechanical control systems of the past are 
replaced by complex linkages and fully-powered or power-boost 
controls, numerous problems concerned with flight control 
system weight, nonlinearities, friction, hysteresis, inertia 
and backlash arise. These problems, together with the increased 
reliance on stability augmentation, have stimulated investi- 
gations of electronic control systems (fly-by-wire). Fly-by- 
wire research has now advanced to the point where test flights 
are being made and favorable and reliable results are being 
reported. (Ref. 1). 

The change from mechanical to electrical control systems 
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offers various possibilities for a cockpit controller that 
is different from the conventional, deck-mounted moveable 
stick. Numerous manipulators have been studied, the most 
prominent of which is a side-located, limited motion hand 
controller (Refs. 2, 3, 4, 5, 6). These systems allow increased 
cockpit space for flight displays or for additional control 
functions and depend, for the most part, on a reliable fly- 
by-wire capability. 

The most important parameter when considering different 
control sticks is that of the aircraft handling qualities. 

The pilot: regards stick feel as a most valuable cue (Ref. 7-)-. 
Due to the irreversibility of power-assisted electronic flight 
control systems, this feel must be provided artificially to 
the stick - whether center or side mounted. Inasmuch as the 
pilot relies heavily on this stick force, the actual motion 
of the controls is of much lesser importance. In fact, it is 
widely recognized that, except possibly during the landing 
flare, the pilot seldom, if ever, knows the position of his 
control stick. (Ref. 4,8). This suggests that a force-only 
(rigid) controller could be applicable to a fly-by-wire con- 
trol system. Such a rigid stick might prove more satisfactory, 
if not for primary control, as a back-up precision tracker to 
be used for formation flying, terrain avoidance, weapon con- 
trol and delivery, carrier landings, or ground controlled 
approaches. Such a back-up system may also prove highly val- 
uable in the case of mechanical failure or combat damage to 
the primary control system. 
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Limited investigations have been made of rigid sticks 
(Refs. 4,9,10,11,12,13,14), but the reported results have 
been contradictatory and essentially inconclusive. 

Inasmuch as handling qualities are inevitably determined 
from pilot opinion, a simulator facility was developed to 
permit the evaluation of rigid control sticks by comparing 
them to similar conventional, moveable sticks. This simu- 
lator, called a Research and Educational Device for Basic 
Aeronautics (RED BARON), employs a two-axis, compensatory 
tracking task with a repeatable, random- appearing signal. 

This investigation used this simulator to measure pilot per- 
formance for one hundred five test subjects on each of four 
control sticks - two moveable and two rigid. This perform- 
ance was compared with individual pilot ratings of the sep- 
arate controllers. Personal pilot experience was collected 
to insure a thorough test subject analysis. Scores were 
also recorded during a portion of the test runs to determine 
the effects on the data, if any, of pilot learning, adaption, 
or fatigue. An approximate human transfer study was conducted 
using two sticks and two subjects for the purpose of correl- 
ation. Additional qualitative comments from the test subjects 
were recorded and a statistical score- to-rating correlation 
study was made. 
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II. SIMULATOR FACILITY 



The simulator facility (RED BARON) designed and built 
for this evaluation has as its major components a Cathode 
Ray Tube display (oscilloscope), an analog computer, 
an electronic counter, a low-frequency function generator, 
a tape deck, a two-channel visual recorder, an electric 
timer, a cockpit environment housing, and interface equip- 
ment. An equipment listing is contained in Appendix D. 

Figure 1 shows the overall view of the simulator cockpit 
environment housing which was constructed on a 96" x 34" 
base on which was mounted a salvaged FJ aircraft ejection 
seat. A cover was constructed over a frame to create an 
aircraft cockpit environment. The CRT display was mounted 
in the windscreen area to give the required visual reference 
for the tracking problem. Figure 2 shows a partial internal 
view of the cockpit with the lower portion of the CRT (with- 
out the cover panel), the pilot's seat and the rigid stick 
visible. 

The tape recorder provides a repeatable testing signal 
which is displayed on an X-Y CRT with a five inch grid. 

The pilot-operated control stick generates a signal which, 
when amplified, acts to cancel out the random input signal 
from the tape, moving the displayed pip towards the center 
of the grid. The control signal is altered in the analog 
computer to simulate actual aircraft dynamics. Thus, in his 
efforts to center the pip on the display, the test subject 
has a constant display of the error signal. 

A ventilator fan, which is activated by the closing of 
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the entrance door, was installed to cool the simulator 
cockpit. In addition, a small fan was mounted in the wind- 
screen area to cool the CRT assembly. 

The test subject could correct for parallax by using 
the horizontal and vertical position knobs on the display 
console (Fig. 3) to center the pip under the mid-grid lines. 
The Airspeed Indicator shown in Figure 3 and a throttle 
assembly were not used during this evaluation. 

Green and blue indicator lights were installed on the 
display console above the CRT (Fig. 3). The green light 
indicates when the target pip is in the scoring area 
(Scorer) and the blue light indicates when the electronic 
counter circuit is energized (Timer). These lights are 
duplicated for the facility operator on the Control and 
Switching Panel (Fig. 4). 

The RED BARON has wiring installed for connection of the 
four control units so that any stick, when plugged in, will 
be connected to the output terminals when the stick selector 
switches on the Control and Switching Panel are in the 
proper position. A toggle switch selects Moveable or Rigid 
stick systems and a four-position rotary switch selects the 
desired control stick. 

The two 12-wire bundles from the stick selector switch 
are identical. One is routed to the side-arm controller 
arm rest and the other goes to the deck-mounted stick area. 
At the end of each wire bundle is a sixteen connector plug 
which has twelve positions (Fig. 5) as follows: 
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1. Common junction of strain gage terminus (longitudinal) 

2. Forward strain gage terminus (longitudinal) 

3. Aft strain gage terminus (longitudinal) 

4. Moveable stick output (longitudinal) 

5. Common junction of strain gages (directional) 

6. Left strain gage terminus (directional) 

7. Right strain gage terminus (directional) 

8. Moveable stick output (directional) 

9. Plus five volts (longitudinal) 

10. Minus five volts (longitudinal) 

11. Plus five volts (directional) 

12. Minus five volts (directional) 

No rudder pedals or facsimiles thereof were installed 
inasmuch as this was to be a two-axis problem and because 
rudder is seldom used in single-engine jet aircraft in the 
cruise configuration. 

An overall view of the facility, including some of the 
circuit wiring between system components is shown in Fig- 
ure 6. 
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III. CONTROL STICKS 



Four different control sticks were constructed for use 
in this evaluation. Two of the sticks were of the conventional 
moveable type using variable potentiometers as signal gen- 
erators. The other two sticks were constructed as rigid types 
using strain gages in a Wheatstone Bridge circuit as the 
signal generators. 

One of the rigid sticks and one of the moveable sticks 
were made a"s deck-mounted types and the other two sticks were 
made into side-arm controller units. 

MOVEABLE DECK-MOUNTED STICK 

The major components of the moveable deck-mounted stick 
were salvaged from a North American FJ aircraft These parts, 
consisting of the stick proper, pitch and roll fulcrums, and 
lever arms were mounted on a plywood base. See Figure 7. The 
height of the stick from the base is 25 5/8" with a moment 
arm of 22" in the pitch direction and a moment arm of 16" in 
the roll (lateral) direction. An artificial feel system was 
installed to develop a stick force in proportion to stick 
displacement simulating the control feel of a jet aircraft. 
This artificial feel is provided by springs mounted in both 
the fore and aft direction and in the lateral direction. No 
bobweights were used. Two variable potentiometers were 
mounted on the control unit, one to generate pitch signals, 
and one to generate lateral signals. The variable potenti- 
ometers are of the one-turn type driven by a 1:4 ratio gear 
drive from the stick. The gearing between the stick position 
and the simulated control surface deflection is a linear 
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relationship even though the majority of powered control 
systems employ a non-linear gearing such that a relatively 
greater stick deflection per control deflection will occur 
at the neutral stick position. Plus and minus five volts 
are the inputs to the potentiometers and the output signals 
are the simulated position indicators of the control surfaces. 
These outputs become the inputs to the switching circuit 
shown in Figure 5, (for clarity, only the moveable hand 
stick circuit is shown) and thence into the analog circuit 
shown in Figure 12. 

The plywood base fits snugly under the cockpit simulator 
seat to provide a solid platform for the operation of the 
stick. The input and output wires are attached to a sixteen 
connector plug which permits the rapid change of control 
sticks . 

MOVEABLE HAND STICK 

The moveable hand stick, as shown in Figure 8, was mounted 
on a quarter inch aluminum box, 3" x 4" x 14” which contains 
the lateral variable potentiometer and the lateral articial 
feel springs. The pitch potentiometer and the pitch feel 
spring are monted externally and forward of the control box. 
The pitch and yaw potentiometers have a 1:4 gear ratio with 
the output from the plus and minus five volt input fed to 
the switching circuit as shown in Figure 12. As in the move- 
able deck-mounted stick, the motion of the hand grip is 
linear in relation to the simulated control surface deflect- 
ion. The control unit is mounted on Velcro fabric for a 
quick change capability with the rigid hand stick. The input 
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and output wires are attached to a sixteen connector plug 
to permit the rapid change of control sticks. 

RIGID DECK-MOUNTED STICK 

The rigid deck-mounted stick was constructed from a 
salvaged helicopter stick, cut down to a size comparable 
to the moveable stick and mounted on an aluminum flange. 

See Figure 9. The flexure, shown in Figure 13a, was machined 
from one-inch diameter 2024-T4 Alcoa aluminum stock. This 
material has an ultimate tensile strength of 68,000 psi 
and a yield strength of 47,000 psi at a temperature of 75° F. 
(Ref. 15). 

It was desired to have the maximum bending stress of the 
flexure approximately one-half of the yield stress in order 
to provide the maximum possible signal, yet to be well 
within the yielding point of the 2024 aluminum. Assuming a 
moment arm of approximately two feet, computed from the top 
third of the control grip to the center of the flexure, and 
a maximum applied force of 15 pounds, the moment becomes 
360 inch-pounds. The formula 

M M 

S = — 

max I bh 2 /6 

was used to calculate the thickness, h = 0.3 inches. An S_.„ 
of 24,000 psi and a 'b' of 1" was used for this calculation. 

Four SR-4 strain gages, type FAB-25-12513 , were attached, 
as shown in Figure 13a, by Eastman 910 cement and then water- 
proofed. The strain gages have a gage factor of 2.07 +_ 1%, 
resistances of 120.0 +_ 0.2 ohms, and were designed to be 
temperature compensated for aluminum. 
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The strain gages were used as two resistances in a 
Wheatstone Bridge that was energized with 10 volts. With 
this circuitry, the bending moment applied will cause the 
voltage changes in the two strain gages to be additive 
while cancelling the effects of a moment applied at right 
angles to the flexure. (Ref. 16). The Wheatstone Bridge 
arrangement is shown in Figure 5. 

The flexure was pressed into a six inch square piece 
of 3/4" aluminum which was then mounted on a plywood base. 

A strain gage guard of three inch aluminum thin-walled 
tubing was installed around the flexure area to protect 
the delicate strain gages and wiring. The attached wires 
were connected to a sixteen connector plug for quick change 
capability . 

RIGID HAND CONTROL STICK 

The rigid hand control stick was mounted on an alum- 
inum control box similar to that of the moveable hand stick 
See Figure 10. The aluminum flexure, as shown in Figure 13b 
was constructed of material identical to that of the rigid 
deck-mounted stick, but the thickness of the flexure was 
reduced to 0.15" which resulted in a maximum stress of 

25.000 psi, computed for a force of fifteen pounds on a 
moment arm of six inches. This compares closely with the 

24.000 psi of the deck stick maximum stress computed using 
a force of fifteen pounds on a 24 inch moment arm. 

Strain gages identical to those used on the deck stick 
were attached and similar wiring, plugs and circuits were 
used. For clarity, Figure 5 shows only the switching and 
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Wheatstone Bridge circuits for the rigid hand stick. 

The hand grips for the hand controllers were made from 
an epoxy mixture of five parts APCO 210 Resin and one part 
APCO 180 Hardener with carbon lampblack added for color. 
The knurled sections of the handles were cast in molds, as 
shown in Figure 11, which had been made using a clay hand 
grip model. 
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IV. ANALOG COMPUTER CIRCUIT 



The inputs to the analog computer are from the Control 
and Switching Panel of the simulator, Figure 4, through a 
patch panel box and a multi-wire extension. The outputs 
from the C & S Panel come from either the variable poten- 
tiometers of the moveable controls or from the output ter- 
minals of the strain gage Wheatstone Bridge circuits of the 
rigid sticks. 

The selection of the stick inputs depends on the switch 
positions on the simulator C & S Panel. In order to change 
these inputs, a toggle switch is used to select either 
moveable or rigid systems and a rotary switch is used to 
select one of the four sticks. The toggle switch provides 
for both changes in the analog input resistors (2,000 ohms 
for rigid sticks, 100,000 ohms for moveable sticks) and 
feedback resistors (400,000 for rigid sticks, 100,000 ohms 
for moveable sticks). These resistor values give an ampli- 
fication factor of one for the moveable and two hundred for 
the rigid sticks. 

The signals are then passed through an additional amplifier 
to increase the amplitude by a factor of ten before the 
signals enter the longitudinal and directional control 
circuits. 

The circuit used to amplify the signals and to simulate 
aircraft response is shown in Figure 11. The longitudinal 
circuit approximates the Short Period motion of an F-4 air- 
craft at 0.9 Mach at sea level. The output is considered to 

be the pitch angle, 0, effected by the dynamic short period 
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mode. Figure 14 shows the analog computer output after a 
step longitudinal input is introduced into the system. In 
the short period approximation, since the airspeed is con- 
stant, the elevator input results in .a 9 change, the mag- 
nitude of which is step input time dependent. In addition, 
the 9 change, as shown in Figure 14, will remain in the 
circuit until removed, due to the lack of airspeed and/or 
altitude change with any elevator input. (Ref. 17). 

The lateral circuit is an approximation of an aileron 
input to a stable aircraft. The response to a step input is 
shown in Figure 15. Reference 18 states that the majority 
of pilots prefer a system where the aileron is bank-ordering 
so that a steady aileron force is required to maintain a 
steady bank angle. In the simulator analog circuit, a step 
input in the lateral mode, as shown in Figure 15, will 
return to the null position after the input is removed 
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V. ANALOG TIMER CIRCUIT 



The scoring principle used in the simulator is based on 
timing the periods when the display pip is within a pre- 
determined scoring area on the CRT display. The test subject, 
using the control stick, attempts to cancel out the pre- 
recorded taped inputs so as to center the pip on the grid. 

The longitudinal error signals and the directional error 
signals are summed independently and then amplified by a 
factor of ten, as shown in Figure 16. The amplified signals 
are passed through a sign changing amplifier, and both the 
original signal and the signal with the reversed sign are 
fed to diodes which allow current flow in only one direction 
when a selected voltage is exceeded. 

The increase in signal magnitude is required to activate 
the diodes which require a minimum of one-half volt before 
passing current. The sign changing amplifiers are necessary 
so that both plus and minus signals will trigger the com- 
parator, which is biased for signals of but one polarity. 

A bias of -0.5 volts is patched to the output side of 
the diode bus so that when any summed, amplified signal 
exceeds this level the diodes will permit current flow to 
the signal comparator input INI terminal. An input of -0.75 
volts is patched to the 1N2 terminal of the comparator to 
provide a comparator bias signal base. The 1N2 input may 
be varied to adjust the size of the CRT display scoring 
area. 

The comparator relay connects the output of a ten hertz 
oscillator to the electronic counter so that when the summed 
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signals (either longitudinal or directional) exceed a speci- 
fied level, the oscillator signal to the counter ceases. 
Since the counter records the ten hertz oscillations, the 
electronic counter recordsthe time that the summed signals 
(both longitudinal and directional) are within the scoring 
area to the nearest tenth of a second. 

A switch on the C & S Panel permits the starting and 
stopping of the counting sequence. 

The comparator relay also activates the Scorer light 
on the cockpit display panel and the C & S Panel to advise 
when the signal pip is within the scoring area. 
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VI. TESTING PROCEDURE 



Before the beginning of each test run, the subjects were 
briefed as to the operation of the simulator and the testing 
plan. Explanations were given as to the nature of the sticks, 
the size of the scoring area, and the control motion and/or 
force required to produce a given pip deflection. In addition, 
the meaning of the Scorer and Timer lights, the testing order 
of the sticks, and the testing run length was explained. 

The test was initiated by selecting the proper stick 
switches on the Control and Switching Panel and turning on 
the tape drive. The first two-minute segment of the tape input 
was a zero signal to permit the balancing of the potentiom- 
eters of the analog circuit (centering the scoring area on 
the grid) and to allow the test subject to correct for par- 
allax by ce nt er i ng the pip^y 

The input signal for the test run was programmed as follows 

1. Two minutes of zero signal. 

2. One minute of longitudinal signal only. 

3. Thirty seconds of zero signal. 

4. One minute of directional signal only. 

5. Thirty seconds of zero signal. 

6. One minute of combined longitudinal and directional 
signal. (Practice run). 

7. Thirty seconds of zero signal. 

8. Three minutes of combined longitudinal and directional 
signal. (Scoring run). 

The test subject was informed when the test signal changed 
modes and was allowed a thirty second rest period (zero sig- 
nal) between each mode Adequate warning was furnished prior 
to the start of the scoring run and the test subject was 
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notified when one minute remained in the scoring run. 

At the end of the scoring run, a changeover switch was 
made to another stick and the testing process was repeated 
until each of the four sticks had been used utilizing the 
identical input signal. 

At the conclusion of the test, the subjects were asked to 
complete a questionnaire evaluating the control sticks and 
providing precise information concerning their total flight 
experience. These evaluations were made before the scores 
were disclosed. The questionnaire used is shown in Table 1. 

In order to make an adequate evaluation of the controllers, 
a suitable rating scale was required. In the past, handling 
qualities research has usually employed the standardized 
Cooper Rating scale (Ref. 19), or, more recently, the mod- 
ified Cooper-Harper scale (Ref. 20). Preliminary testing 
during a "shake-down” period disclosed that a finer discrim- 
ination was needed than was available with the standardized 
scales. Therefore, a new rating scale was devised (Table 2) 
which allows for a wider range of "satisfactory" ratings, 
while still providing the opportunity for assigning an ad- 
verse opinion. 

The preliminary testing also disclosed a tendency for the 
test subject's performance to improve slightly as the test 
progressed from one stick to another. For this reason, the 
testing order was varied in an effort to cancel, or at least 
minimize, the learning function effects. Inasmuch as the pri- 
mary purpose of the study was to compare rigid to moveable 
sticks, the variation of test order was concerned only with 
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TABLE 1 



PILOT QUEST IONAIRRE 

Name Test Number 

Age Date 



FLIGHT EXPERIENCE 

Approximate Number of Pilot Hours in: 

Single-Engine Jet 

Multi-Engine Jet 

Single-Engine Prop 

Multi-Engine Prop 

Helicopter 

Light General A/C 

Non-Pilot 

Type of Operational A/C With Most Experience (F-4, A-4, etc.) 



Type of Operational A/C With Most Recent Experience. 



How Long Since Piloted Any Aircraft? (If over one month) 



How Long Since on Full-Time Operational Flight Status? 



PILOT OPINION AND PERFORMANCE 

Pilot Rating Test Score Preference 

Moveable Center Stick 

Moveable Side-Arm Stick 

Rigid Center Stick 

Rigid Side-Arm Stick 
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TABLE 2 



PILOT RATING SCALE 

(Numerical Rating - Descriptive Phrases) 

1. Fantastic, could not be improved. Should be in all A/C. 

2. Excellent control response, no gripes. 

3. Good response, pleasant to fly. 

4. Good response, would require some getting used to. 

5. Satisfactory response, would expect no difficulty. 

6. Satisfactory, would expect minor problems at times 

7. Acceptable, but with some unpleasant characteristics. 

8. Unacceptable for normal operations 

9. Unacceptable for any operations. 

10. Unsatisfactory, dangerous, uncontrollable. 
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alternating the rigid and moveable pairs In each pair of 
systems, the center-mounted stick was used first. 

To determine any possible learning, adaption or fatigue 
effects during the tests, partial scores were observed and 
recorded at thirty second intervals for thirteen of the test 
subjects. Finally, qualitative comments by the test subj'ects 
concerning the validity of the simulation and the applica- 
bility of the rigid stick concept were encouraged. 
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VII. ANALYSIS AND CLASSIFICATION OF TEST SUBJECTS 



To properly interpret the data in a test of this type, it 
is necessary to have background information on the test sub- 
jects. The flight experience data obtained from the test 
subjects' questionnaires are shown in tabular form in Appen- 
dix A. 

Included in the one hundred five test subj’ects were Naval 
Aviators, Naval Flight Officers, private pilots and non- 
pilots representing a broad spectrum of aircraft flight 
experience. The private pilots and non-pilots tested can 
be considered, in general, as representative of the type of 
personnel who enter Naval flight training. 

PILOT CLASSIFICATION 

To facilitate an interesting and meaningful evaluation of 
the scores and ratings, the test subj'ects were classified 



into five major groups: 

1. Jet (JET) Pilots 31.0% 

2. Propeller (PROP) Pilots 24.2% 

3. Helicopter (HELO) Pilots 15.5% 

4. Private (PRI) Pilots 13.8% 

5. Non-pilots (NON) 15.5% 



It can be seen that a relatively even distribution was 
obtained, there being enough subj'ects in each category to 
insure an accurate data base. Three subj'ects had significant 
experience in two different categories and were therefore 
included in both. The classification of each subject is shown 
in Appendix B. 
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This division of skills was made for several reasons. 

First, it was felt necessary to evaluate the opinions and 
performance of the subjects in light of their previous 
experience. Different types of pilots use different control 
sticks. In general, the jet and helicopter pilots are exper- 
ienced with the conventional, deck-mounted moveable control 
stick, while propeller and private pilots use yoke or wheel 
controllers. Single engine propeller aircraft pilots do use 
the center-mounted control stick, but only two subjects 
(former T-28 flight instructors) had the majority of their 
experience in aircraft of this type. Second, pilot class- 
ification of the data could provide some indication of the 
type of aircraft in which a rigid stick might be most applic- 
able. For example, if the jet pilots preferred a rigid stick 
system, this type might be more readily installed in jet 
aircraft. Finally, the performance and evaluation of non- 
pilots as compared with that of the pilots is required to 
determine the effect, if any, of long-established flying 
tradition on the opinions of the pilots. It should be mentioned, 
however, that all of the non-pilots did have some limited 
amount of flight experience, although in most cases this was 
less than twenty hours. 

TEST SUBJECT ANALYSIS 

The average age of the test subjects was 27.9 years, with 
67.3% between 27 and 32 years of age. A significant differ- 
ence of 5 to 12 years was noted between the age of the non- 
pilots (younger) and the pilots 
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80% of the pilots had flown some type of aircraft with- 
in one month of the day they were tested (Combat Readiness 
Training flying) The average time since any flight was 2.7 
months, including five subjects who had not flown for over 
a year. In general, the private pilots had not flown recently. 

The Navy and Marine pilots who were tested were students 
at the Naval Postgraduate School and not on operational 
flight status. The average time since these pilots had been 
on operational flight status was 13 months. 58.7% of the 
pilots had been on operational status between 5 and 12 
months prior to their test Only nine of the pilots had 
not flown operationally in over 2% years. 

A wide variety of Naval aircraft was represented in the 
experience bank of the military pilots. These aircraft 
included : 



JET: 


A-4, 


A- 6 , A- 7 , F-4, F-8 


PROP: 


A- 1 , 


C- 121 , P-2, P-3, P-5, S-2, T-28 


HELO: 


H-2 , 


H-4 , H-24, SH-3 



63% of the military pilots had their most recent exper- 
ience in the same type of aircraft in which they had had 
the most experience. Only one pilot had significant recent 
experience in an aircraft of a different type (a Helo pilot 
in an S-2 ) . 

The average military pilot tested had flown 2200 total 
pilot hours with 86.8% having between 1000 and 4000 pilot 
hours. The private pilots had significantly less flight 
experience, with an average of 200 hours apiece. 
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In general, it may be said that the majority of those 
tested were moderately experienced military pilots, about 
30 years of age, who had been on operational flight status 
one year previous, and were currently making proficiency 
flights in the same generic type of aircraft (Jet, Helo , et 
cetera) at the time of the tests. 
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VIII. TEST RESULTS 



By the end of the testing program, results were obtained 
from one hundred five subjects, including eighteen non- 
pilots. The data obtained -- test scores, pilot ratings, 
and testing order -- are given in Appendix C. Several 
methods of data reduction and comparison were utilized for 
a complete and meaningful evaluation of the test results. 
Average scores and ratings were computed for each pilot 
classification and for the entire test group. Since averages 
can give an incomplete, and often erroneous, view, the raw 
scoring and pilot rating distributions are given to sup- 
plement the averaged performance and rating of opinions. 

In addition to the comparison of the various control sticks, 
the individual pilot classifications are compared to further 
illuminate the subject. 

AVERAGES 

The over-all average test score for each stick (out of 
a possible total of 180 seconds) is shown in Figure 21. It 
is evident that the body of the test subjects did better 
with the rigid controllers than with the moveable sticks. 

The best performance was on the rigid side-arm stick, while 
the poorest by far was with the moveable hand controller. The 
average score with the moveable center-mounted stick, while 
lower, compared favorably with the rigid systems. 

Figure 32 presents the average pilot rating for each 
stick. The rigid sticks were preferred overwhelmingly over 
the moveable controls The force-only sticks were found to 
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have "good response", while the moveable controllers were 
rated only "satisfactory". The subjects tended to dislike 
the moveable side-arm controller, and little difference 
was noted between the two rigid sticks. At least on the 
average, pilot opinion correlated with actual performance. 

A more detailed score- to-rating correlation study is in- 
cluded in Section IX. 

COMPARISON OF STICKS 

More detailed comparisons of the four control sticks 
for particular pilot classes are shown in Figures 19 and 
20. These results (Fig. 19) show that all groups did better 
with the rigid controls to varying degrees. It can also be 
seen that Jet, Helicopter and Non-pilots exhibit variations 
in performance identical to those of the overall averages 
(Fig. 21); i.e., rigid side-arm -- best, and moveable side- 
arm -- worst. Notable differences are: (1) the Prop pilots 

did slightly better with the rigid center-mounted stick 
than with the rigid hand controller, and (2) private pilots 
scored higher with the moveable side stick than with the 
deck-mounted moveable stick. 

It can be seen from Figure 20 that all pilot groups 
preferred the rigid to the moveable controls, and disliked 
the moveable side stick. Opinion was fairly evenly divided 
as to which stick was the best -- rigid center-mounted or 
rigid side-arm. The relative magnitudes of all the rating 
averages are about the same, indicating that the pilots' 
interpretations of the rating scale was quite uniform. 
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COMPARISON OF PILOT CLASSES 



Actual comparison of the different pilot groups for a 
particular control stick is shown in Figures 22 and 23 This 
analysis reveals that private pilots and non-pilots scored 
higher than pilots with the rigid control sticks. As expected, 
jet pilots performed better than the other classes with the 
moveable center-mounted stick due to their previous experience 
with this type of control The results also indicate that 
propeller pilots, in general, encounter difficulty with the 
sidearm controllers, and both prop and helo pilots tended 
to score below the average on all sticks. 

SCORE AND RATING DISTRIBUTION 

Charts of the test score and pilot rating distributions 
were made as a part of the averaging analysis The over-all 
performance distribution plot of Figure 24 shows the per- 
centage of the total number of test subjects whose final 
score lay in each five-second time interval throughout the 
range of test scores. A statistical distribution of scores 
about the mean is approached with the two rigid sticks and 
the moveable center-mounted stick (Fig. 24a,b,c). It is to 
be noted that the moveable side-arm controller results are 
marked by a rather even distribution of scores concentrated 
at the lower end of the scale. 

The over-all pilot opinions (Fig. 25) are somewhat more 
divided on the moveable systems than on the rigid sticks -- 
especially in the case of the moveable hand controller, where 
the ratings are particularly scattered. 

Indentical performance and pilot opinion distribution 
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plots for the different pilot classifications are included 
in Appendix E. Facts of interest from these graphs are: (1) 

Jet pilot scores on the moveable side stick were divided 
into two groupings -- one average and one poor (Fig. E-ld); 
(2) Two individual prop pilots scored extremely low on the 
rigid side-arm controller -- lowering the over-all perform- 
ance average for this classification (Fig. E-3b); and (3) 
While the scores of the private pilots were concentrated, 
their opinions of all sticks were scattered -- indicating 
that perhaps not enough private pilots were tested for a 
valid comparison (Figs. E-7, E-8). 
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IX. HUMAN FACTORS INVOLVEMENT WITH TEST VALIDITY 



In a test program of this nature, dealing with variations 
in pilot ability and the vagaries of personal opinion, a 
great deal of care must be taken to insure validity of the 
data. Human involvement errors can be introduced by myriad 
differences in adaption, learning, fatigue and experience, 
and by fluctuations in attention, motivation and judgement. 
Safeguards must be established and checks need to be made 
to cancel or minimize these possible sources of error. 

Several methods of analysis were utilized to determine 
the nature of human involvement. A score- to-rating correla- 
tion study was conducted to determine the effects, if any, 
of pilot bias. A standard regression analysis was applied 
to the data as a means of representing each score and 
rating on a single graph. Learning, adaption and fatigue 
of the test subjects was examined by observing the variation 
of performance throughout a run, and the testing order of 
the controllers was varied throughout the tests. Additional 
information concerning learning and adaption, as well as 
variations in stick dynamics, was sought by an approximate 
human transfer function study with the simulator. A check 
was also made to learn if there was a variation in test 
results if the magnitude of the pip deflection were changed. 
This was necessary inasmuch as the pip scale may have been 
altered during the seven month testing program. Finally, the 
realism of the simulation was studied by comparing non- 
pilot performance to that of pilots and by soliciting 
individual comments from the pilots who were tested 
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SCORE-TO-RATING CORRELATION 



With the introduction of a new device like a rigid 
control stick, the possibility exists that a pilot could 
become enhanced with its novelty and evaluate it accordingly. 
On the other hand, a pilot with many hours experience with 
a moveable deck-mounted stick might tend to dislike the 
rigid controls, even though his performance is superior 
while using them. To determine the magnitude of this type 
of problem in the testing program, a score- to-opinion cor- 
relation analysis was applied. Plotting the over-all average 
scores versus the average ratings for each stick as shown 
in Figure 26 gives a first approximation as to how well the 
pilot opinions correlate with performance. The strong cor- 
relation is obvious, since the rating becomes rapidly less 
favorable (higher number) with decreasing test scores. 
Similarly, in Figure 27, a regression analysis, plotting 
score versus rating and reducing the points to a single 
line for each controller, shows this strong correlation 
of performance to pilot evaluation. 

For a more detailed analysis, an individual correlation 
factor "r" was calculated for each test subject. These 
factors are listed in Table 3, with a perfect correlation 
being r = 1.000. This value of r is derived from the square 
root of the square to avoid the negative number that would 
appear due to the fact that the rating scale is inverted. 

That is to say, a decreasing pilot rating occurs with an 
increasing rating number. From Table 3 it can be seen that 
fifty-six of the one hundred five subjects have a high 
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TABLE 3 



INDIVIDUAL PILOT SCORE-TO- RATING CORRELATION FACTORS 



Subject 


r 


Sub j ec t 


r 


Sub j ec t 


r 


Subject 


r 


Subject 


r 


' 1 


. 912 


22 


.394 


43 


.000 


64 


.935 


85 


. 690 


2 


.978 


23 


.546 


44 


.918 


65 


.515 


86 


.952 


3 


. 718 


24 


. 829 


45 


.941 


66 


.929 


87 


.880 


4 


.972 


25 


.971 


46 


. 669 


67 


. 835 


88 


.446 


5 


. 350 


26 


. 642 


47 


. 900 


68 


.733 


89 


.387 


6 


.971 


27 


. 932 


48 


. 993 


69 


. 933 


90 


. 670 


7 


.460 


28 


.952 


49 


. 758 


70 


. 956 


91 


. 752 


8 


.539 


29 


.994 


50 


.007 


71 


.954 


92 


.325 


9 


. 619 


30 


. 985 


51 


. 914 


72 


.951 


93 


.971 


10 


.911 


31 


.965 


52 


. 260 


73 


.749 


94 


.975 


11 


.660 


32 


.926 


53 


. 004 


74 


.717 


95 


.873 


12 


.697 


33 


.900 


54 


. 532 


75 


.935 


96 


.795 


13 


. 978 


34 


. 204 


55 


.866 


76 


. 979 


97 


.875 


14 


.767 


35 


.453 


56 


. 741 


77 


.826 


98 


.529 


15 


.804 


36 


. 576 


57 


. 796 


78 


.430 


99 


. 873 


16 


.746 


37 


.426 


58 


.812 


79 


. 886 


100 


.990 


17 


. 948 


38 


.952 


59 


. 984 


80 


. 981 


101 


. 702 


18 


.833 


39 


. 149 


60 


.493 


81 


. 952 


102 


. 945 


19 


.967 


40 


. 109 


61 


.307 


82 


. 892 


103 


.427 


20 


. 206 


41 


. 990 


62 


. 381 


83 


. 844 


104 


. 595 


21 


.982 


42 


.803 


63 


.572 


84 


.976 


105 


.976 



38 



correlation of .800 or better. Twenty-five percent of the 
subjects tested have a poor score- to-correlation (below 
.400). A close examination of the scores and ratings of 
these subjects reveals the causes of this poor correlation. 
Twelve of these testees had poor correlation due to their 
nearly identical scores on the separate sticks. The remain- 
ing poor correlations can be laid, at least in part, to 
pilot bias, as evidenced from the comments of these subjects. 

The average correlation for each pilot classification is 
of interest and is given in Figure 28. As expected, the 
less-experienced private and non-pilots had significantly 
lower score-to-rating correlations than those of the military 
pilots. 

Thus, with a high over-all average correlation of .680, 
it appears that pilot bias had little effect on the data. 

The majority of the test subjects had relatively high 
performance- to-opinion correlations. The poor correlations 
were few in number, and the errors introduced tended to be 
mutually cancelling. 

LEARNING, ADAPTION, AND FATIGUE. 

During the preliminary testing it was noted that the 
test subject's skill appeared to improve as the testing 
progressed. To avoid the possibility of one type of stick 
obtaining an advantage over the other, the testing order 
(moveable versus rigid) was changed for each subject. 
Fifty-three subjects used the rigid controls first and the 
remaining fifty-two started with the moveable sticks. Thus 
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any possible learning function effects should have been 

nearly eliminated. Evaluating the increase in score from 

the first stick used to the next, the average learning 

was of the order of one second (0.56% of the maximum possible) 

and thus essentially negligible. 

An almost universal comment from the test subjects upon 
finishing the test was that the tracking task was extremely 
tiring. To ascertain further learning, adaption or fatigue 
factors, partial test scores were recorded at 30-second 
intervals for thirteen test subjects. The results obtained 
from this survey are shown in Figure 29. It is evident that 
the test scores dropped off rather rapidly during the final 
30-second time frame This could be due to the fact that 
the subjects were given a notification one-minute prior to 
the end of the run. Motivation may have lowered slightly 
in anticipation of the completion of a somewhat tedious task. 

Little evidence of slow adaption to the controllers can 
be found from the results. The subjects in general appear 
to have become fully adapted to each stick during the alloted 
practice time that preceeded each scoring run. The scores 
in the first frames show only a slight improvement in per- 
formance, with actually a decrease while using the moveable 
side-arm controller. This is not to be considered as being 
too surprising since human operator control adaption usually 
occurs in one to three seconds following a change in simple 
tracking conditions (Ref. 21). 

An effort was made to compute an approximate human transfer 
function for two test subjects using the RED BARON facility. 
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It was hoped that a more analytical description of human 
adaption and learning could be found (Ref. 22). The subjects 
completed a sinusoidal tracking task in the directional 
mode at various frequencies of oscillation, using the move- 
able center-mounted stick and the rigid side-arm controller. 
Their frequency responses were recorded and are illustrated 
in the Bode plots of Figures 30 and 31. Pilot transfer 
functions were obtained from these plots by the asymptotic 
approximation method. The similar form of the curves are an 
indication that human response, and thus adaption, is com- 
parable for a given stick; but that the response to two 
different sticks can be quite diverse. No further effort was 
made to apply these results to the data since the study was 
but approximate and since indications of adaptive difficulty 
in the tests were absent 
SCORING PIP DEFLECTION 

Since the testing program extended over a several month 
period, the motion of the pip on the CRT display for a given 
control force or displacement could quite possibly have been 
changed. This could have occurred due to alternate usage of 
the tape deck or actual tape deterioration. Periodic calibra- 
tions of the pip were made, but the possibility still existed 
that the pip motion could have been altered between the 
calibrations. To check this source of error, three subjects 
were tested with two different pip deflections -- one larger 
and one smaller than the norm. As might be expected, the 
scores with a small deflection were higher as the pip was 
easier to control. However, little change was found in the 
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relative differences between the four control sticks. Thus 
it was assumed that even if small changes occurred in the 
pip deflection, the error, for comparison purposes, was 
minimal . 

TEST LIMITATIONS AND SIMULATION VALIDITY 

Most pilots seemed to agree that while their scores and 
ratings were higher with the rigid systems in the simulator, 
the situation might be significantly different in an actual 
aircraft. Herein lies the major limitation to an opinion 
survey of this sort. It was felt by most of the pilots that 
actual flight testing would be necessary to determine, for 
certain, pilot acceptance or rejection of a rigid control 
stick. The relatively high scores of the non-pilot group 
cast a further doubt on the realism of the simulation. The 
age differential between the non-pilots and the pilots could 
well be a factor in this score spread, however. 

Another key limitation concerns the low force gradients 
incorporated into all four sticks. The maximum force necessary 
for full scale deflection of the pip was about one pound, 
while the optimum stick force per g in an actual aircraft 
is five to seven pounds (Ref. 23). This low force was de- 
liberately introduced into this phase of the testing. It 
was desired to. have the moveable controls nearly force 
independent so that they would be nearly a motion-only type 
of system. The rigid controls were rigged for a force that 
approximated that of the moveable controls for purposes of 
comparison. A more thorough and comprehensive study should 
include the effects of variations of stick force and stick 
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force gradients per pip deflection and deflection rate on 
pilot opinion and performance A throttle and airspeed indi- 
cator have been rigged for this purpose, with the throttle 
controlling the feedback resistances of the analog computer 
so that increased throttle (as evidenced by increased airspeed 
indications) results in increased force requirements. This 
system was not energized during this phase of the testing 
in order that all those tested could be compared at the same 
base . 

This study was further limited in that the relative effects 
of vibration were ignored. Aircraft vibration can reduce the 
manual dexterity of the pilot and introduce an additional 
unsteadiness in his control motion. The human body tends to 
damp out vibration; thus problems could arise with the side- 
arm controllers where the pilot's arm rests on a surface 
that would vibrate with the aircraft. Reference 1 reported 
that a moveable control stick gave performance superior to 
the rigid controller at all exciting frequencies of vibration 
tested. Vibration would naturally enter a flight test program 
and could also be introduced into a simulator. An interesting 
sidelight of vibration effects is the ascertion by several 
neuro-muscular experts that there is a low frequency neuro- 
muscular response that, if in resonance with external vib- 
rations, could incapitate the control function of the human 
operator . 

The effects of acceleration on control stick performance 
should also be investigated in a flight test situation rather 
than in a ground-based simulator. 



43 



During the course of the testing, several pilots expressed 
concern about the effect on a rigid controller of movement of 
conventional stick-mounted trim switches and microphone 
buttons. While these switches normally create no stick motion 
when operated, they would involve the addition of an extra 
force on a rigid control stick. Unless unconventional switches 
were installed, this would mean applying an opposite compen- 
sating force when using the trim tabs, to avoid undesired 
control deflection. This problem could be included in further 
simulator studies. 

A final limitation might lie in the fact that the subjects 
tested were not trained test pilots as is the custom in 
handling qualities simulator studies. The limited qualitative 
nature of the information requested in the evaluation of the 
sticks, however, should have relieved this requirement, and 
the high score-to-rating correlation achieved by the test 
subjects supports this assumption. In addition it was desired 
to determine the opinions of average fleet pilots with regard 
to the acceptance of a rigid cockpit control system. 
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X. DISCUSSION AND CONCLUSIONS 



The purpose of this investigation was to determine the 
acceptability to operational military pilots of a rigid 
cockpit control system and to evaluate their performance 
while using force-only controls. Within the restrictive 
framework of the test limitations mentioned in Section IX, 
this has effectively been accomplished. Results were obtained 
from one hundred five flying and non-flying test subjects 
operating both moveable and rigid controllers in a compensa- 
tory tracking task. 

To draw specific conclusions from the collected data, it 
is necessary to thoroughly analyze the test subject group 
in order to ascertain the applicability of the results. At 
the same time, particular measurement standards and procedures 
must be reviewed to insure validity of the test. Finally, in 
a study involving human opinions, the possible sources of 
error induced by the subjects and inherent in the test itself 
must be examined. 

The test subject group consisted of Navy and Marine aviators, 
Naval Flight Officers, private pilots and non-pilots. The 
pilots who were tested were on proficiency flying status and 
therefore may not have been quite as sharp as their active 
fleet counterparts. The non-pilot group was highly represnt- 
ative of the personnel who enter Naval flight training. A 
significant number of each pilot classification -- jet, prop, 
helo, private and non-pilot -- was tested to provide a sound 
base for significant conclusions. The possible exception was 
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the private pilot class which showed basically erratic 
performance and ratings, possibly due to the duration of 
time since many of them had flown, and their relatively low 
flight experience factor. In general, however, the inexper- 
ienced subjects were included in the testing in order to 
compare their performance with that of the pilots and the 
opinions of the inexperienced groups were of less import- 
ance to the study. The wide variety of aircraft included 
in the experience of the military pilots represented a 
majority of the operational aircraft in the fleet today. 

The measurement procedures used in this investigation 
were relatively conventional and the test validity was 
protected. The use of signal error in a compensatory track- 
ing task is a common measure of pilot performance and is a 
fairly standard procedure. The revised rating scale, allow- 
ing for finer discriminations, was especially devised for 
this study. It success is evidenced by the relatively 
uniform interpretations it received by the various test 
subject classifications. 

Numerous precise measurements were conducted to eliminate 

errors introduced by the human test subjects and the test 

apparatus. The extensive correlation analysis effectively 

ruled out test subject bias in most cases. It demonstrated 

that the small amount of bias found tended to cancel itself 

out; i.e., although some pilots preferred the moveable 

system and scored high with the rigid, others preferred 

the rigid and scored high with the moveable controls. The 

number of negative correlations was few. Alternating the 

testing order of the control slicks virtually eliminated 
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pilot learning effect during the course of the test. Even 
so, the learning value factor was a very small number. Most 
of the test subjects experienced fatigue during the final 
30 seconds of the run; however, this only introduced a 
difference of about one-half of one percent in the over-all 
test score. The fatigue was slightly greater with the move- 
able controls -- another indication of their difficulty. No 
problem was experienced with test subject adaptation to a 
particular control stick. The tape segment alloted to 
practice and familiarization allowed time for nearly complete 
adaptation. The error due to any possible change in pip 
deflection was negligible, since the relative differences 
of scores and ratings for each stick remained essentially 
constant. These various safeguards and checks greatly min- 
imized the various possible sources of error to the data. 

In general, the pilots performed better with the rigid 

control sticks and preferred them to the moveable systems. 

The rigid side-arm controller was the consensus favorite in 
both performance and preference. The moveable side-arm stick 
was uniformly disliked and its performance was inferior. The 
moveable deck-mounted stick was only slightly less preferred 
by the pilots than the rigid systems -- the differences in 
scores and ratings were not large; whereas the less exper- 
ienced private and non-pilots displayed a rather marked 
preference of rigid over moveable controllers with a similar 
large difference in performance with each system. 

These results, as a preliminary indication, suggest that 
a rigid controller is certainly feasible either as a primary 
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control, back-up control, or a precision tracker in an 
aircraft with a fly-by-wire capability 

It would not be economically feasible to replace present 
moveable controls with force-only sticks, since the difference 
in indicated performance and present -pilot rating is not 
large. Therefore, on the basis of the preliminary results 
of this study, the rigid control systems appear worthy of 
further investigation for future primary control systems 
or present secondary tracking functions. 

The major simulation limitations lie in low stick force 
gradients, lack of vibration (as is found in aircraft), and 
the absence of acceleration effects. This study should not 
be considered as a blanket indictment of the moveable side- 
arm controller. It is extremely possible that the dynamics 
of the particular model tested contributed to a great degree 
to its low rating Gravity effects were not included, but 
there is a possibility that a rigid control system may be 
superior under high g conditions, due to the difficulties 
of gross arm motions as would be required with a moveable 
control . 

All of these limitations could seriously affect the pilot 
performance and acceptance of rigid control systems as re- 
ported in this study. However, the derived results indicate 
a preliminary feasibility, with a strong requirement for 
future investigation and flight testing. 
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Figure 1. External Overall View of Simulator 




Figure 2. Internal View of Simulator Cockpit 
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Figure 3. Display Console 




Figure 4 . Control and Switching Panel 
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RIGID STICK 



-MOVABLE STICK 




FIGURE 5. 

WHEATSTONE BRIDGE AND SWITCHING CIRCUIT 
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Figure 6. Overall View of Simulator Facility 




Figure 7. Moveable Deck-mounted Stick 
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Figure 8. Moveable Hand Control Stick 




Figure 9. Rigid Deck-mounted Stick 
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Figure 10 Rigid Hand Control Stick 




Figure 11. Hand-Grip Model and Molds 
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FIGURE |3^ 

RIGID STICK FLEXURE INSTALLATION 
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FIGURE 13b. 

RIGID HAND STICK FLEXURE 
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INSTALLATION 
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Figure 14. Analog Computer Response (Longitudinal) 




Figure 15. Analog Computer Response (Directional) 
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FIGURE 16 . 
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Figure 17. Typical Scoring Run (Longitudinal) 




Figure 18. Typical Scoring Run (Directional) 
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Figure 20. Average Rating By Each Pilot Classification 
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Figure 21. Average Test Score With Each Stick 

- All Subjects - 
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Figure 31. Pilot Frequency Response with Moveable Center Stick 
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APPENDIX A 



LIST OF EQUIPMENT 

1. Analog Computer 

Pace TR-10 

Electronic Associates, Inc. 

2. Timer (Scorer) 

Universal EPUT and Timer, Model 7350 
Berkeley Division, Beckman Co. 

3. Timer 

Lab-Chron 

Laboratories Industries, Inc. 

4. Oscilloscope 

Model 120A 
Hewlett-Packard Co. 

5. Visual Recorder 

Two-Channel, Model BL-52U 
Brush Electronics Co. 

6. Low Frequency Function Generator 

Model 202A 
Hewlett-Packard Co. 

7. Power Supply (2) 

Model 3569 

Systems Research Corp . 

8. Tape Deck 

One-half inch, Model FR-100A 
Ampex Corp. 
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APPENDIX B 
(Continued ) 



£ 

o 

H 

H 

< 

U 

M 

ft. 

H 

(/) 

CO 



< 

u 



CL o OOOCLOOuCLOOO 

H 

ft) 



H I I I O I I I I 1 

OJ l 00 

Oh III I I I I I 



I 

I 



o 

IT) 

CM 



I 

I 



I o 

00 
I r-f 



I 



o 

_) 

ft) 



Oh 

ft) 

s 



a. 

ft) 

co 



i i i i i i i 

i i i i i i i 



o 

I O i i I I I 

r— I 

I CM I I I I I 



I o o o o o o 
in in in o o 

I CM CM ^ CM CM r-l 



O i 


i o O l 


1 1 O 1 1 


1 1 i o 


i o 


i o 


in 


O CM 


o 


o 


o 


o 


*<* i 


l CM 00 l 


1 1 VO 1 l 


i i i in 


i o 


1 CM 




r— 1 


r-l 


r— 1 


CM 


r— 1 



o 


i O I 


1 o 


l O l I 


1 1 


in 


o 


o 


O 




C 1 ) 


I in i 


i CM 


I ^ I i 


1 1 




CO 


r— 1 


CM 





o o 


i o o o o 


o o o 


i o o o 


i o o o 


I o 


l o 


in o 


in o in o 


in o o 


in in o 


in m vo 


m 


in 


CM CM 


1 N N H O 


CM in CM 


l CO N Tf 


1 H H H 


l CM 


1 CM 



CM 






o 

o 



I O l l i I I 

in 

I r-l I I I I I 



I i I o 




I i i o 



1 



o 

o 



•“) i i 


i o o o o o 


1 O 1 


i O 1 


i O i i 


I 1 O O 1 


I I o 


ft) 


o o m o o 


o 


o 


o 


o o 


o 


CO 1 1 


i o m h oo o 


1 CM 1 


1 CM 1 


i O 1 i 


i 1 CO M0 l 


i i in 




r-l r-l r-l 


CM 


r— 1 


r-l 


CM 


CM 



ft) I 

o GONvOCMOOt^O^OCOCM'OHNOOO'OONCOOOONOCMOOOCOCMOOOO 
< I CMCMCMCOCMCMCOCMCOCMCMCMCOCMCOCOCOCMCMCMCMCOCMCMCMCMCMCMCMCM 



H 

a 

■-> 

CO 

CO 



Hc^co^m^NcoooHojcn^in^NcooOHCMcn^in^Nco^o 

OvO\OvO\0\OvOvOvONNNNNC-NNNhOOOOOOCOCOOOOOOOCOCOO> 



78 



APPENDIX B 
(Continued ) 
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PILOT SCORING AND RATING DATA 
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APPENDIX C 
(Continued ) 
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APPENDIX C 
(Continued ) 
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